Traditionally convolutional neural network architectures have been designed by stacking layers on top of each other to form deeper hierarchical networks. The cortex in the brain however does not just stack layers as done in standard convolution neural networks, instead different regions are organized next to each other in a large single sheet of neurons. Biological neurons self organize to form topographic maps, where neurons encoding similar stimuli group together to form logical clusters. Here we propose new self-organization principles that allow for the formation of hierarchical cortical regions (i.e.

Our empirical evaluation across sequential image classification and language modelling tasks shows that subLSTM units can achieve similar performance to LSTM units.

First provide a summary of the paper, and then address the following criteria: Quality, clarity, originality and significance. Summary: The authors present a model of auto-associative memory in a rate-based neural network subject to a battery of biological plausible constraints. Previous models of auto-associative memory have failed to include several key features of real biological networks, namely an adherence to Dale's Law that neurons have a strictly excitatory or inhibitory effect on their projections and the observation that networks can encode memories without relying on units that simply respond at their saturation rate or respond in a binary manner. Memories are encoded in the network via synaptic modifications based on a gradient descent procedure, constrained using a recently published method for ensuring that the linearization of the dynamics around a dynamical system's fixed point is stable. The authors illustrate the effectiveness of their training procedure with simulations, noting that the trained fixed points exhibit slow network dynamics (i.e. they are close to being, but are not exactly, fixed points) and are stable, as desired.

The persistent and graded activity often observed in cortical circuits is sometimes seen as a signature of autoassociative retrieval of memories stored earlier in synaptic efficacies. However, despite decades of theoretical work on the subject, the mechanisms that support the storage and retrieval of memories remain unclear. Previous proposals concerning the dynamics of memory networks have fallen short of incorporating some key physiological constraints in a unified way. Specifically, some models violate Dale's law (i.e.

Summary: This paper introduces a new learning framework in leaky integrate and fire neurons, which permits a recurrent network to efficiently learn linear dynamical systems. The approach uses weight changes at two timescales: fast weight changes quickly balance excitation and inhibition, while slower weight changes learn the structure of the LDS. A key insight is that the fast plasticity which balances excitation and inhibition distributes a global signal about the network's performance to all neurons, enabling error driven learning of the LDS with a local learning rule. Major comments: This paper presents the intriguing idea of using the balance of excitation and inhibition to distribute global error information throughout a neural network, permitting supervised learning with a local learning rule. Moreover, the scheme introduced is based on predictive coding, which as the paper shows, naturally leads to sparse irregular spiking activity. On this subtle view, neural firing in response to an identical input will not yield identical precise spike times; but the particular spike times for each input presentation are nonetheless precisely arranged, and cannot be replaced by a rate coded approximation without a drop in fidelity or efficiency.

To predict sensory inputs or control motor trajectories, the brain must constantly learn temporal dynamics based on error feedback. However, it remains unclear how such supervised learning is implemented in biological neural networks. Learning in recurrent spiking networks is notoriously difficult because local changes in connectivity may have an unpredictable effect on the global dynamics. The most commonly used learning rules, such as temporal back-propagation, are not local and thus not biologically plausible. Furthermore, reproducing the Poisson-like statistics of neural responses requires the use of networks with balanced excitation and inhibition.

Neural Information Processing Systems http://nips.cc/

The persistent and graded activity often observed in cortical circuits is sometimes seen as a signature of autoassociative retrieval of memories stored earlier in synaptic efficacies. However, despite decades of theoretical work on the subject, the mechanisms that support the storage and retrieval of memories remain unclear. Previous proposals concerning the dynamics of memory networks have fallen short of incorporating some key physiological constraints in a unified way. Specifically, some models violate Dale's law (i.e.

How do neural networks learn to represent information? Here, we address this question by assuming that neural networks seek to generate an optimal population representation for a fixed linear decoder. We define a loss function for the quality of the population read-out and derive the dynamical equations for both neurons and synapses from the requirement to minimize this loss. The dynamical equations yield a network of integrate-and-fire neurons undergoing Hebbian plasticity. We show that, through learning, initially regular and highly correlated spike trains evolve towards Poisson-distributed and independent spike trains with much lower firing rates.

Summary: The SLK protein plays a key role in neuron excitability and sensitivity, researchers report. Nerve cells can regulate their sensitivity to incoming signals autonomously. A new study led by the University of Bonn has now discovered a mechanism that does just that. The German Center for Neurodegenerative Diseases and the Max Planck Institute for Neurobiology of Behavior was involved in the work. The results have now been published in the journal Cell Reports.